Pause the Video: Quick But Quantitative Expert Evaluation of Tutorial Choices in a Reading Tutor that Listens

To critique Project LISTEN’s automated Reading Tutor, we adapted a panel-of-judges methodology for evaluating expert systems. Three professional elementary educators watched 15 video clips of the Reading Tutor listening to second and third graders read aloud. Each expert chose which of 10 interventions to make in each situation. To keep the Reading Tutor’s choice from influencing the expert, we paused each video clip just before the Reading Tutor intervened. After the expert responded, we played back what the Reading Tutor had actually done. The expert then rated its intervention compared to hers. Although the experts seldom agreed, they rated the Reading Tutor’s choices as better than their own in 5% of the cases, equally good in 36%, worse but OK in 41%, and inappropriate in only 19%. The lack of agreement and the surprisingly favorable ratings together suggest that either the Reading Tutor’s choices were better than we thought, the experts knew less than we hoped, or the clips showed less than they should.